Goal-oriented control of a robotic arm

ABSTRACT

Methods and systems for improved control of robotic arms are presented. In one embodiment, a method is presented that includes predefining a plurality of motion primitives, which may include one or more preconditions and effects. A target state for a plurality of workpieces may be determined as well as an initial state of the plurality of workpieces. A sequence of operations may be generated based on the preconditions and/or effects of the motion primitives, as well as the target state and the initial state. Executing the sequence of operations may be capable of changing the plurality of workpieces from the initial state to the target state.

BACKGROUND

The use of robotic arms has impacted many industries over the years. Therobotic arms can vary in functionality, ranging from the ability tocomplete simple motions to performing complex tasks, and the samerobotic arm may be used for multiple types of tasks. Such motions andtasks require programming of the robotic arm to perform the actions in aparticular sequence depending on the specified motion or task.

SUMMARY

The present disclosure presents new and innovative systems and methodsfor controlling the movement of robotic arms. In a first aspect, amethod for controlling a robotic arm to manipulate a plurality ofworkpieces is provided that includes predefining a plurality of motionprimitives of the robotic arm, each of the plurality of motionprimitives comprising one or more preconditions and one or more effects,and acquiring a target state of the plurality of workpieces. The methodmay further include acquiring an initial state of the plurality ofworkpieces and generating a sequence of operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state and the initial state of theplurality of workpieces. The operations may be selected from theplurality of motion primitives and execution of the sequence ofoperations may be capable of changing the plurality of workpieces fromthe initial state to the target state.

In a second aspect according to the first aspect, acquiring the targetstate of the plurality of workpieces includes identifying a state of aprototype which corresponds to the target state of the plurality ofworkpieces.

In a third aspect according to the second aspect, the state of theprototype includes relative positions and relative connections betweenworkpieces of the prototype.

In a fourth aspect according to any of the second and third aspects, thestate of the prototype is identified with a vision acquisition device.

In a fifth aspect according to any of the first through fourth aspects,acquiring the initial state of the plurality of workpieces includesacquiring the initial state of the plurality of workpieces with a visionacquisition device.

In a sixth aspect according to any of the first through fifth aspects,the initial state of the plurality of workpieces comprises locations ofthe plurality of workpieces, and the sequence of operations is generatedbased on the locations of the plurality of workpieces.

In a seventh aspect according to any of the first through sixth aspects,the initial state of the plurality of workpieces includes relativepositions and relative connections between the plurality of workpieces,and the sequence of operations is generated based on the relativepositions and relative connections between the plurality of workpieces.

In an eighth aspect according to any of the first through seventhaspects, generating the sequence of the operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state and the initial state of theplurality of workpieces includes generating the sequence of theoperations in such a way that: every precondition of a start one of theoperations meets the initial state of the plurality of workpieces andevery effect of a last one of the operations meets the target state ofthe plurality of workpieces.

In a ninth aspect according to the eighth aspect, generating thesequence of the operations based on the one or more preconditions andthe one or more effects of the plurality of motion primitives, thetarget state and the initial state of the plurality of workpiecesfurther includes generating the sequence of the operations in such a waythat: one or more effects of a previous one of the operations is capableof changing the plurality of workpieces from a previous state to asubsequent state. One or more preconditions of a subsequent one of theoperations may meet the subsequent state.

In a tenth aspect according to any of the eighth and ninth aspects, thesequence of the operations is generated by using a graph traversalalgorithm.

In an eleventh aspect according to any of the first through tenthaspects, the method further includes performing a collision check todetermine whether the sequence of the operations is feasible and,responsive to determining that the sequence of the operations is notfeasible, regenerating a new sequence of operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state and the initial state of theplurality of workpieces.

In a twelfth aspect according to any of the first through eleventhaspect, the method further includes performing a collision check todetermine whether the sequence of the operations is feasible and,responsive to determining that the sequence of the operations isfeasible, executing the sequence of operations.

In a thirteenth aspect, a system for controlling a robotic arm isprovided that includes a processor and a memory storing instructionswhich, when executed by the processor, cause the processor to perform amethod for controlling the robotic arm to manipulate a plurality ofworkpieces. The method may include predefining a plurality of motionprimitives of the robotic arm, each of the plurality of motionprimitives comprising one or more preconditions and one or more effects,and acquiring a target state of the plurality of workpieces. The methodmay further include acquiring an initial state of the plurality ofworkpieces and generating a sequence of operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state and the initial state of theplurality of workpieces. The operations may be selected from theplurality of motion primitives and execution of the sequence ofoperations may be capable of changing the plurality of workpieces fromthe initial state to the target state.

In a fourteenth aspect according to the thirteenth aspect, the systemfurther includes a vision acquisition device. The state of the prototypemay be identified with the vision acquisition device and the initialstate of the plurality of workpieces may be identified with the visionacquisition device.

In a fifteenth aspect according to any of the thirteenth and fourteenthaspects, the initial state of the plurality of workpieces includeslocations of the plurality of workpieces, and relative positions andrelative connections between the plurality of workpieces. The sequenceof operations may also be generated based on the locations of theplurality of workpieces, and the relative positions and the relativeconnections between the plurality of workpieces.

In a sixteenth aspect according to any of the thirteenth throughfifteenth aspects, generating the sequence of the operations based onthe one or more preconditions and the one or more effects of theplurality of motion primitives, the target state and the initial stateof the plurality of workpieces includes generating the sequence of theoperations in such a way that: every precondition of a start one of theoperations meets the initial state of the plurality of workpieces; andevery effect of a last one of the operations meets the target state ofthe plurality of workpieces; and one or more effects of a previous oneof the operations is capable of changing the plurality of workpiecesfrom a previous state to a subsequent state, wherein one or morepreconditions of a subsequent one of the operations meet the subsequentstate.

In a seventeenth aspect according to any of the thirteenth throughsixteenth aspects, the method further includes performing collisioncheck to determine whether the sequence of the operations is feasibleand responsive to determining that the sequence of the operations is notfeasible: regenerating a new sequence of operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state and the initial state of theplurality of workpieces.

In an eighteenth aspect, a non-transitory, computer readable medium isprovided storing instructions which, when executed by a processor, causethe processor to perform a method for controlling the robotic arm tomanipulate a plurality of workpieces. The method may include predefininga plurality of motion primitives of the robotic arm, each of theplurality of motion primitives comprising one or more preconditions andone or more effects, and acquiring a target state of the plurality ofworkpieces. The method may also include acquiring an initial state ofthe plurality of workpieces and generating a sequence of operationsbased on the one or more preconditions and the one or more effects ofthe plurality of motion primitives, the target state and the initialstate of the plurality of workpieces. The operations may be selectedfrom the plurality of motion primitives and execution of the sequence ofoperations may be capable of changing the plurality of workpieces fromthe initial state to the target state.

In a nineteenth aspect according to the eighteenth aspect, generatingthe sequence of the operations based on the one or more preconditionsand the one or more effects of the plurality of motion primitives, thetarget state and the initial state of the plurality of workpiecesincludes generating the sequence of the operations in such a way thatevery precondition of a start one of the operations meets the initialstate of the plurality of workpieces; and every effect of a last one ofthe operations meets the target state of the plurality of workpieces;and one or more effects of a previous one of the operations is capableof changing the plurality of workpieces from a previous state to asubsequent state, wherein one or more preconditions of a subsequent oneof the operations meet the subsequent state.

In a twentieth aspect according to any of the eighteenth and nineteenthaspects, the state of the prototype includes relative positions andrelative connections between workpieces of the prototype. The initialstate of the plurality of workpieces may also include locations of theplurality of workpieces, and relative positions and relative connectionsbetween the plurality of workpieces. The sequence of operations may alsobe generated based on the relative positions and relative connectionsbetween workpieces of the prototype, the locations of the plurality ofworkpieces, and the relative positions and the relative connectionsbetween the plurality of workpieces.

The features and advantages described herein are not all-inclusive and,in particular, many additional features and advantages will be apparentto one of ordinary skill in the art in view of the figures anddescription. Moreover, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and not to limit the scope of the inventivesubject matter.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a system according to an exemplary embodiment of thepresent disclosure.

FIGS. 2A-2B illustrate motion primitives according to exemplaryembodiments of the present disclosure.

FIG. 3 illustrates views of a workpiece arrangement according to anexemplary embodiment of the present disclosure.

FIG. 4 illustrates a method for assembling and disassembling workpiecesaccording to an exemplary embodiment of the present disclosure.

FIG. 5 illustrates a method for creating and storing motion primitivesaccording to an exemplary embodiment of the present disclosure.

FIG. 6 illustrates a robotic arm according to an exemplary embodiment ofthe present disclosure.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

When a robotic arm is programmed to perform a task, the actionsperformed by the robot and the sequence in which the robotic armexecutes the actions may be extremely important to the overallperformance of the task. For example, when constructing an assembly frommultiple parts, the order in which the parts are connected may beimportant for ensuring proper operation of the assembly. Correctperformance of tasks by robotic arms may typically require operators tomanually program particular actions and the particular sequence for theactions depending on the specified task. However, a task may not alwaysbe performed identically each time. For example, changes in position forthe parts may alter the specific actions and/or sequence in which theactions are performed (e.g., speed, location, size, shape, etc.).Therefore, the robotic arm may have to be programmed with everypotential sequence of actions that may be required to perform the taskin multiple different scenarios and/or may have to be programmed toaccount for variables which may change.

Proper performance of the robotic arm may also require an operator incertain instances. The operator may monitor conditions within theenvironment in which the robotic arm is placed in order to ensure thatthe robotic arm is not impeded in its completion of the task. As eachtask may be unique, the operator may be required to constantly inputdifferent commands to safeguard against potential mistakes by therobotic arm. This can be incredibly difficult and may require numerousoperators to regularly supervise operations and vary the coding of therobotic arm. Additionally, the operator may need to manually constructor otherwise program the series of actions that the robotic arm mustperform in order to add a new function to the robotic arm (e.g., toprocess a new workpiece) and/or to account for a new variable (e.g., anew arrangement of workpieces) that may alter the sequence of actions tobe performed. Adding functionality in this way can be unduly tedious, asthe operator may be required to specify which actions have to beperformed by the robotic arm. Therefore, there exists a need to enable arobotic arm to accurately and efficiently construct a series of actionsto accomplish new tasks and/or to respond to new environmentalvariables.

One solution to this problem is to configure the robotic arm toautomatically generate a sequence of operations in order to perform aspecified task. These operations may be selected from predefined motionprimitives which specify the actions the robot is capable of taking.Rather than requiring an operator to repeatedly construct actions forthe robotic arm to execute in performing the task, the robotic arm maybe able to select a series of motion primitives to generate the sequenceof operations in order to perform the specified task. The robotic armmay take into account variables prior to selecting motion primitives.For example, the robotic arm may be communicatively coupled to a visualsensory system that can identify information regarding the surroundingenvironment and the workpieces that may be handled by the robotic arm.By taking into account different variables such as the informationidentified by the visual sensory system and information regarding thecurrent state of the robotic arm, the robotic arm can select anefficient series of motion primitives to accomplish the goal.

FIG. 1 illustrates a system 100 according to an exemplary embodiment ofthe present disclosure. The system 100 may be configured to analyzeinformation regarding an environment to select a series of motionprimitives to generate a sequence of operations to perform a specifiedtask. The system 100 includes a computing device 102, which may beconfigured to process information about one or more workpieces and thesurrounding environment prior to selecting motion primitives to beperformed.

The computing device 102 includes a processor 104, a memory 105, and acontroller 106. The computing device 102 may be configured to perform avariety of functions. The processor 104 and the memory 105 may beconfigured to perform one or more of the functions of the computingdevice 102. For example, the memory 105 may store instructions which,when executed by the processor 104, cause the processor 104 to performone or more operational features of the computing device 102. The memory105 may also store information from the performance of the system 100,including information of the environment and a history of taskscompleted by the system 100. The computing device 102 may be configuredto detect and process information regarding an environment. For example,the computing device 102 and the visual sensory device 116 mayconstitute a visual sensory system which may identify one or moreworkpieces 108 in the environment and may identify information regardingthe workpieces 108, such as positions of the workpieces 108 and relativeconnections between the workpieces 108. For example, the workpieces 108can have position information that instructs the computing device of theworkpieces' 108 current orientation and location relative to oneanother. One workpiece may be connected to other workpieces, and theconnections between workpieces 108 can also be identified by the visualsensory system such that the system 100 may know whether the workpieces108 are connected to each other and the connection types between theworkpieces 108. Other information regarding the workpieces 108 (e.g.,shapes, sizes, colors) may be input by an operator of the system 100 andstored in the memory 105 such that the visual sensory system canidentify each workpiece 108.

The information of the workpieces 108 can be gathered through a visualsensory device 116. The visual sensory device 116 may include one ormore cameras or other visual sensors. Data (e.g., image or other visualdata) from the visual sensory device 116 may be analyzed to identifyworkpieces 108 and related information (e.g., positions 110 and/orrelative connections 112). The visual sensory device 116 may becommunicatively coupled to the computing device 102 to send informationabout the workpieces 108 to the computing device 102 for processing bythe processor 104. For example, the computing device 102 may receiveimage data from the visual sensory device 116 and may analyze the imagedata to identify the workpiece 108 and related information. The visualsensory device 116 may also be attached to the robotic arm 114 for anaccurate determination of positions of the workpieces 108 in relation tothe robotic arm 114. Additionally or alternatively, certain relatedinformation for the workpiece 108 may be specified by an operator (e.g.,an operator specifying the task which the robotic arm 114 is beingprogrammed to perform).

The robotic arm 114 may be configured to perform a variety of tasksdepending on the motion primitives 120 selected by the computing device102. Motion primitives 120 may be stored in a database 118, which may becommunicatively coupled to the computing device 102 and the robotic arm114. The database 118 may be configured to receive new motion primitives120 created by a user or transmitted from other system or device. Themotion primitives 120 stored in the database 118 may specify all or atleast part of actions that the robotic arm 114 is capable of performing.Each of the motion primitives 120 may contain one or more preconditions122 that are required prior to the motion primitive 120 being carriedout by the robotic arm 114. Each of the motion primitives 120 may alsocontain one or more effects 124 that indicate the consequent changes tothe state of workpieces 108 and the robotic arm 114 brought bycompleting the motion primitive 120. For instance, one exemplary motionprimitive may be opening the gripper. Its preconditions 122 may includethat the gripper is closed, and its effect 124 may include that theobject which was held by the gripper is no longer held by the gripper.In certain implementations, the robotic arm 114 may receive commandsfrom the computing device 102. For example, the computing device 102 mayprovide commands to the robotic arm 114 based on the motion primitives120 selected from the database. In further implementations, the commandsmay be transmitted from the computing device 102 to the robotic arm 114via the controller 106, which may be communicatively coupled to therobotic arm 114.

FIGS. 2A and 2B depict exemplary motion primitives 200, 210 that may beused to control the operation of a robotic arm. The motion primitive 200may be performed to move a workpiece and the motion primitive 210 may beperformed to screw in a workpiece. In particular, the motion primitives200, 210 may be exemplary implementations of motion primitives 120stored in the database 118 and may be utilized by the computing device102 to generate a sequence of operations to control the motion of therobotic arm 114. The motion primitives 200, 210 include preconditions202, 204, 212, 214, 216, operations 206, 208, 218, 220, and effects 222,224, 226, 228. For example, the motion primitive 200 contains twopreconditions 202, 204. The preconditions 202, 204 may specify one ormore conditions or statuses that must be fulfilled before the motionprimitive 200 can be performed. In particular, the precondition 202 mayrequire that the robotic arm's 114 gripper is currently grasping aworkpiece 108 and the precondition 204 may require that a path ofmovement (e.g., a path of movement for the workpiece 108) isunobstructed. The computing device 102 may be configured to determinewhether the preconditions 202, 204 are met. For example, in order todetermine whether the preconditions 202, 204 are met, the visual sensorydevice 116 may determine an initial state of the workpiece 108 bycollecting and sending image data of, e.g., the workpiece 108, the pathof movement, and/or the robotic arm's 114 gripper to the computingdevice 102. As another example, the system 100 may determine whether theprecondition 202 is met using sensors (e.g., force sensors) in a gripperor in joints of the robotic arm 114. As a specific example, thecomputing device 102 may determine that the gripper is grasping theworkpiece 108 if a grip force exceeds a predetermined threshold. Inanother embodiment, the computing device 102 may not require the inputof data from the visual sensory system 116 or the sensors of the roboticarm 114. In such embodiments, the computing device 102 may rely oninformation stored in the memory 105. For example, operations performedby the system 100 may be stored in the memory 105. The computing device102 may access the memory 105 to determine if a precondition 202, 204 ismet based on previously performed operations. In still furtherimplementations, the preconditions 202, 204 may be verified based on theeffects of earlier motion primitives.

If the computing device 102 determines that all preconditions 122 aremet, performance of the operations 206, 208 may be permitted. Motionprimitive 200 contains two operations 206, 208. The performance ofoperations 206, 208 may be required to complete motion primitive 200. Inmotion primitive 200, the computing device 102 may permit theperformance of operation 206 of moving the workpiece 108 to a particulardestination. Operation 208 may then be performed to release the gripperof the robotic arm 114. As the workpiece 108 has been moved to thedesired destination through operation 206, the computing device 102 mayinstruct the robotic arm 114 to release the workpiece 108 by releasingthe gripper. After completing the motion primitive 200, one or moreeffects 222, 228 may be enacted. In particular, after completing motionprimitive 200, the workpiece 108 is not grasped, as indicated by theeffect 222, and the workpiece 108 is moved to the particulardestination, as indicated by the effect 228.

FIG. 2B depicts a second exemplary motion primitive 210. Motionprimitive 210 demonstrates the operations 217, 218, 220 that may beexecuted by the robotic arm 114 to screw in a workpiece 108 (e.g., tomount the workpiece 108 using a threaded connection). Motion primitive210 contains three preconditions 212, 214, 216. All three preconditions212, 214, 216 may be required to be fulfilled before the operations 217,218, 220 can be executed. In particular, the precondition 212 mayrequire that the robotic arm's 114 gripper is currently grasping aworkpiece 108. Precondition 214 requires that the workpiece 108 ispositioned over a separate parent workpiece. Precondition 216 requiresthat the connection types of the workpieces 108 (e.g., the workpiece 108gripped by the robotic arm 114 and the parent workpiece) are compatible.In order to determine whether the precondition 216 is met, the visualsensory device 116 may collect and send image data of the workpieces 108to the computing device 102. Workpieces 108 may have a particularconnection type. For example, a workpiece may have a connection typesuch as a male and/or female connector (e.g., threaded/screw connection,plug/outlet connection, clamps, etc.). As workpieces 108 may have aparticular connection type, the computing device 102 may need to analyzethe image data sent from the visual sensory device 116 to determine theparticular connection type of each workpiece 108. To meet precondition216, the computing device 102 may use the information of the connectiontypes of each workpiece 108 to determine if the workpieces 108 arecompatible. For example, compatible workpieces 108 may include aworkpiece 108 with a threaded male connector and a parent workpiece witha threaded opening, or female connector. An example of incompatibleworkpieces 108 may include a workpiece 108 with a threaded maleconnector and a parent workpiece with a threaded male connector. Theconnection type of at least some of the workpieces 108 may be inputtedby an operator of the robotic arm 114 and stored in the memory 105 ofthe computing device 102 in advance instead of using the visual sensorydevice 116.

If the computing device 102 determines that the preconditions 212, 214,216 of motion primitive 210 are met, performance of operations 217, 218,220 may be permitted. Due to the preconditions 212, 214, 216 being met,the robotic arm 114 may be gripping the workpiece 108 and the workpiece108 may be positioned over a parent workpiece that the relativeconnections 112 are aligned and compatible. Motion primitive 210contains three operations 217, 218, 220. For example, the performance ofoperations 217, 218, 220 may be required to complete motion primitive210. In particular, the operation 217 may require that the robotic arm114 move the workpiece 108 to contact the parent workpiece. For example,the computing device 102 may instruct the robotic arm 114 to move thegripped workpiece 108 towards the parent workpiece until the connectingportions of both workpieces are in contact. Operation 218 may includerotating the gripper of the robotic arm 114 until a threshold torquelevel is reached, screwing the workpiece 108 into the parent workpiece.Operation 220 may include releasing the workpiece 108 by the gripper ofthe robotic arm 114. After completing the motion primitive 210, one ormore effects 224, 226 may be in place. In particular, after completingmotion primitive 200, the workpiece 108 is not grasped, as indicated bythe effect 222, and the workpiece 108 is mounted on the parentworkpiece, as indicated by the effect 226.

In certain implementations, one or both of the preconditions 202, 204,212, 214, 216 and the effects 222, 224, 226, 228 may be stored aslogical expressions. For example, the preconditions 202, 212 may bestored as WorkpieceGrasped, the precondition 204 may be stored as NOTPathObstructed, and the precondition 214 may be stored as Workpiece1OVER Workpiece2. As another example, the effect 228 of motion primitive200 may be stored as WorkpieceMoved and the effect 226 of motionprimitive 210 may be stored as Workpiece1 ON Workpiece2.

FIG. 3 depicts views 320, 330, 340 of a workpiece arrangement 300according to an exemplary embodiment of the present disclosure. The view320 may be a frontal view of the workpiece arrangement 300 and the views330, 340 may respectively be side views to the left and right of theview 320. The workpiece arrangement 300 includes workpieces 304, 306,308, 310 of different sizes and dimensions arranged on a base 302. Inparticular, the workpieces 304, 306 are positioned directly on the base302 (e.g., the base 302 is the “parent” of the workpieces 304, 306). Theworkpiece 308 is positioned on top of the workpiece 306 (e.g., theworkpiece 306 is the parent of the workpiece 308) and the workpiece 310is positioned on the workpieces 304, 308 (e.g., the workpieces 304, 308are both parents of the workpiece 310). The workpieces 304, 306, 308,310 and the base 302 may be fitted or otherwise attached together (e.g.,via clasps, insertion mounts and the like) or may be positioned asdepicted without being attached together. In certain instances, theworkpiece arrangement 300 may be represented as a logical expression,such as (workpiece 306 ON base 302) && (workpiece 304 ON base 302) &&(workpiece 308 ON workpiece 306) && (workpiece 310 ON workpiece 308) &&(workpiece 310 ON workpiece 304).

FIG. 4 demonstrates a method 420 according to an exemplary embodiment ofthe system 100. The method 420 may be performed to control a robotic arm114 to perform a desired task. For example, the method 420 may beperformed to accomplish a goal or process with regards to the assemblyor disassembly of workpieces through predefined motion primitives. Themethod 420 may be implemented on a computer system, such as the system100. The method may also be implemented by a set of instructions storedon a computer readable medium that, when executed by a processor, causethe computer system to perform the method 420. For example, all or partof the method 420 may be implemented by the processor 104 and the memory105. Although the examples below are described with reference to theflowchart illustrated in FIG. 4 , many other methods of performing actsassociated with FIG. 4 may be used. For example, the order of some ofthe blocks may be changed, certain blocks may be combined with otherblocks, one or more of the blocks may be repeated, and some of theblocks described may be optional.

The method 420 may begin with predefining motion primitives 120 (block422). The motion primitives 120 may be stored in the memory 105 of thesystem 100. The motion primitives 120 may be accessed by the computingdevice 102 in order to accomplish a goal or desired process. The motionprimitives 120 may correspond to operations to be performed by thesystem 100. As explained above, the motion primitives 120 may includepreconditions 122 and/or effects 124. Creating motion primitives isdiscussed in greater detail in connection with the method 500 below.

The system 100 may then acquire a target state of the workpieces 108(block 424). For example, the goal or desired process may include themovement, reordering, or alteration of one or more workpieces 108 to atarget state. The target state may include positions and connections ofthe one or more workpieces 108 with respect to other workpieces 108. Thetarget state may be inputted directly by an operator. For example, theoperator may enter that workpieces 108 need to be moved to a particularlocation and/or positioned with particular orientations relative to oneanother. Alternatively, the target state of a workpiece may be acquiredthrough the use of the visual sensory device 116. For example, thesystem 100 may be presented with a prototype or particular arrangementof workpieces 108 which the system 100 must replicate. The computingdevice 102 may instruct the visual sensory device 116 to acquire imagedata of the workpieces 108, including position 110 and/or relativeconnection 112. The visual sensory device 116 may send the image data tothe computing device 102 for processing by the processor 104. Theprocessor 104 may determine the target states for each workpiece 108.The information regarding the target states may be stored in the memory105. In one particular example, the target state of the workpieces 108may be the workpiece arrangement 300.

Data regarding initial states of the workpieces 108 may then bedetermined (block 426). For example, after acquiring the target statesof the workpieces 108, the system 100 may be presented with workpieces108 which are in initial states. Although the goal may be to move ororient the workpieces 108 in a particular manner corresponding to thetarget state, the workpieces 108 may begin in a location and position110 and/or orientation that is different than the target state. Thecomputing device 102 may receive the data regarding the initial statesof the workpieces 108 through use of the visual sensory device 116. Forexample, the visual sensory device 116 may acquire image data of thecurrent set up of the workpieces 108, denoting the initial states of theworkpieces 108. The visual sensory device 116 may send the image data ofthe initial states to the computing device 102. The computing device 102may process the image data using the processor 104. Alternatively, theinitial states of the workpieces 108 may be entered by an operator. Inan example where the target state is the workpiece arrangement 300, theinitial state may indicate that the workpieces 304, 306, 308, 310 areall connected to the base 302 (e.g., at locations spread out from oneanother). In certain instances, the initial states may includeorientations of the workpieces 108 relative to one another. Continuingthe previous example, the initial state may indicate that the workpieces310, 308 are perpendicular to the workpiece 306 (rather than parallel asin the workpiece arrangement 300) and that the workpiece 304 is parallelto the workpiece 306 (rather than perpendicular as in the workpiecearrangement 300).

Based on the initial states and target states of the workpieces 108, thesystem 100 may generate a sequence of corresponding operations toaccomplish a goal and reach the target states of the workpieces 108(block 428). The sequence of operations may be further based on motionprimitives 120, preconditions 122, and effects 124. For example, thecomputing device 102 may select motion primitives 120 based on thetarget states of the workpieces 108 and the initial states of theworkpieces. In particular, the computing device 102 may select thecorresponding operations such that the initial state meets thepreconditions 122 of at least the first operation and such that the endresult of the effects 124 of the sequence of operations meets the targetstate of the workpieces 108. In certain implementations, the effects 124of completing a first operation may result in the fulfillment of aprecondition 122 for a later, second operation. In particular,completing the first operation may transition the workpieces 108 from aprevious state to a subsequent state, where the previous state does notcomply with at least one of the preconditions 122 of the secondoperation and the subsequent state complies with every precondition 122of the second operation. In certain implementations, the sequence ofoperations may be identified at least in part using a graph traversalalgorithm, such as the A* search algorithm or the like. In particular,such a search may be performed on a database 118 of motion primitives120 to identify the sequence of operations. Continuing the example wherethe target state for the workpieces 108 is the workpiece arrangement300, the sequence of operations may include (1) picking up the workpiece304 and placing it on the base 302 next to the workpiece 306, (2)picking up the workpiece 308 and placing it on the workpiece 306, and(3) picking up the workpiece 310 and placing it on the workpieces 304,308.

The computing device 102 may execute the sequence of operations (block430). For example, the computing device 102 may execute the operationsin an order specified by the sequence of operations (e.g., by executingone or more motion steps of motion primitives included in the sequenceof operations). In certain instances, prior to executing the sequence ofoperations, the computing device 102 may perform a collision checkanalysis for the feasibility of successfully completing the sequence ofoperations. For example, the computing device 102 may analyze thesequence of operations to determine whether particular workpieces arenot accessible (e.g., because other workpieces or equipment block therobotic arm 114 from accessing the workpieces 108). As another example,the computing device 102 may analyze the sequence of operations todetermine whether performing particular operations (e.g. movingworkpieces 108) will cause the workpieces 108 to collide with otherequipment or workpieces. If the computing device 102 determines that acollision is likely, or that the sequence of operations is otherwise notfeasible, a new sequence of operations may be generated (e.g., byrepeating block 428).

FIG. 5 demonstrates a method 500 according to an exemplary embodiment ofthe system 100. The method 500 may be performed to program and store amotion primitive 508. For example, the method 500 may be performed tostore motion primitives in the database 118 for use in the method 420(e.g., during block 422). The method 500 may be implemented on acomputer system, such as the system 100. The method 500 may also beimplemented by a set of instructions stored on a computer readablemedium that, when executed by a processor, cause the computer system toperform the method 500. For example, all or part of the method 500 maybe implemented by the processor 104 and the memory 105.

The method 500 may begin with receiving one or more motion steps 502.For example, the computing device 102 may receive one or more motionsteps 502 that are to be performed when executing the motion primitive508. The motion steps 502 may specify actions and/or movements performedby the robotic arm 114. The method 500 may also include receiving one ormore preconditions 504 and one or more effects 506. The preconditions504 and the effects 506 may be received in connection with the motionsteps 502. For example, the preconditions 504, effects 506, and/ormotion steps 502 may be received from a user. As a specific example, theuser may use the computing device 102 or another computing device tocreate and/or enter the motion steps 502, the preconditions 504 and theeffects 506.

A motion primitive 508 may then be created. For example, the computingdevice 102 may create a motion primitive 508 that includes the motionsteps 502, the preconditions 504 and the effects 506. The motionprimitive 508 may then store the motion primitive 508 in the database510, which may be an exemplary implementation of the database 118. Forexample, the computing device 102 may transmit the motion primitive 508to the database 510 for storage. Additionally or alternative, the motionprimitive 508 may be stored in a database 510 implemented within thememory 105 of the computing device 102.

FIG. 6 depicts an exemplary robotic arm 600. The robotic arm 600 may becommunicatively coupled with the system 100 and may be an exemplaryimplementation of the robotic 114. The robotic arm 600 may includemultiple robotic arm segments 602 and an end effector 604. The endeffector 604 may include a gripper, in which one or more sensors (e.g.,force and/or torque sensors) may be installed. For example, the endeffector 604 may be a robot gripper that utilizes the sensor(s) todetect changes in force and/or torque as described above in connectionwith operations 217, 218. In other examples, the end effector 604 may beany other suitable tool that utilizes any suitable sensor(s) asdescribed in the present disclosure. In some embodiments, the sensor(s)to detect changes in force and/or torque may be alternatively oradditionally be installed in the joint actuator of the robotic armsegments 602.

All of the disclosed methods and procedures described in this disclosurecan be implemented using one or more computer programs or components.These components may be provided as a series of computer instructions onany conventional computer readable medium or machine readable medium,including volatile and non-volatile memory, such as RAM, ROM, flashmemory, magnetic or optical disks, optical memory, or other storagemedia. The instructions may be provided as software or firmware, and maybe implemented in whole or in part in hardware components such as ASICs,FPGAs, DSPs, or any other similar devices. The instructions may beconfigured to be executed by one or more processors, which whenexecuting the series of computer instructions, performs or facilitatesthe performance of all or part of the disclosed methods and procedures.

It should be understood that various changes and modifications to theexamples described here will be apparent to those skilled in the art.Such changes and modifications can be made without departing from thespirit and scope of the present subject matter and without diminishingits intended advantages. It is therefore intended that such changes andmodifications be covered by the appended claims.

The invention claimed is:
 1. A method for controlling a robotic arm tomanipulate a plurality of workpieces, comprising: predefining aplurality of motion primitives of the robotic arm, each of the pluralityof motion primitives comprising one or more preconditions and one ormore effects; acquiring a target state of the plurality of workpieces,wherein the target state comprises relative connections between at leasta subset of the plurality of workpieces; acquiring an initial state ofthe plurality of workpieces, wherein the initial state comprisesconnection types for at least the subset of the plurality of workpieces;determining if the connection types for at least the subset of theplurality of workpieces are compatible with each other beforeperformance of operations; and generating a sequence of operations basedon the one or more preconditions and the one or more effects of theplurality of motion primitives, the target state and the initial stateof the plurality of workpieces, wherein generating the sequence of theoperations is performed in a way that every precondition of a startingone of the operations meets the initial state of the plurality ofworkpieces and every effect of a last one of the operations meets thetarget state of the plurality of workpieces; wherein the initial stateof the plurality of workpieces further comprises relative positions andrelative connections between the plurality of workpieces, and thesequence of operations is generated based on the relative positions andthe relative connections between the plurality of workpieces in theinitial state; wherein the operations are selected from the plurality ofmotion primitives; wherein execution of the sequence of operations iscapable of changing the plurality of workpieces from the initial stateto the target state, including connecting at least the subset of theplurality of workpieces according to the relative connections in thetarget state and the connection types.
 2. The method of claim 1, whereinthe acquiring the target state of the plurality of workpieces comprises:identifying a state of a prototype which corresponds to the target stateof the plurality of workpieces.
 3. The method of claim 2, wherein thestate of the prototype comprises relative positions and relativeconnections between workpieces of the prototype.
 4. The method of claim2, wherein the state of the prototype is identified with a visionacquisition device.
 5. The method of claim 1, wherein acquiring theinitial state of the plurality of workpieces comprises acquiring theinitial state of the plurality of workpieces with a vision acquisitiondevice.
 6. The method of claim 1, wherein the initial state of theplurality of workpieces comprises locations of the plurality ofworkpieces, and the sequence of operations is generated based on thelocations of the plurality of workpieces.
 7. The method of claim 1,wherein the generating the sequence of the operations based on the oneor more preconditions and the one or more effects of the plurality ofmotion primitives, the target state, and the initial state of theplurality of workpieces further comprises: generating the sequence ofthe operations in such a way that: one or more effects of a previous oneof the operations is capable of changing the plurality of workpiecesfrom a previous state to a subsequent state, wherein one or morepreconditions of a subsequent one of the operations meet the subsequentstate.
 8. The method of claim 1, wherein the sequence of the operationsis generated by using a graph traversal algorithm.
 9. The method ofclaim 1, further comprising: performing a collision check to determinewhether the sequence of the operations is feasible; and responsive todetermining that the sequence of the operations is not feasible:regenerating a new sequence of operations based on the one or morepreconditions and the one or more effects of the plurality of motionprimitives, the target state, and the initial state of the plurality ofworkpieces.
 10. The method of claim 1, further comprising: performingcollision check to determine whether the sequence of the operations isfeasible; and responsive to determining that the sequence of theoperations is feasible: executing the sequence of the operations.
 11. Asystem for controlling a robotic arm, comprising: a processor; and amemory storing instructions which, when executed by the processor, causethe processor to perform a method for controlling the robotic arm tomanipulate a plurality of workpieces, wherein the method comprises:predefining a plurality of motion primitives of the robotic arm, each ofthe plurality of motion primitives comprising one or more preconditionsand one or more effects; acquiring a target state of the plurality ofworkpieces, wherein the target state comprises relative connectionsbetween at least a subset of the plurality of workpieces; acquiring aninitial state of the plurality of workpieces, wherein the initial statecomprises connection types for at least the subset of the plurality ofworkpieces; determining if the connection types for at least the subsetof the plurality of workpieces are compatible with each other beforeperformance of operations; and generating a sequence of operations basedon the one or more preconditions and the one or more effects of theplurality of motion primitives, the target state and the initial stateof the plurality of workpieces, wherein generating the sequence of theoperations is performed in a way that every precondition of a startingone of the operations meets the initial state of the plurality ofworkpieces and every effect of a last one of the operations meets thetarget state of the plurality of workpieces; wherein the initial stateof the plurality of workpieces further comprises locations of theplurality of workpieces, and relative positions and relative connectionsbetween the plurality of workpieces; wherein, the sequence of operationsis generated based on the locations of the plurality of workpieces, andthe relative positions and the relative connections between theplurality of workpieces in the initial state; wherein the operations areselected from the plurality of motion primitives; wherein execution ofthe sequence of operations is capable of changing the plurality ofworkpieces from the initial state to the target state, includingconnecting at least the subset of the plurality of workpieces accordingto the relative connections and the connection types.
 12. The system ofclaim 11, further comprising a vision acquisition device, wherein astate of a prototype corresponding to the target state is identifiedwith the vision acquisition device; and the initial state of theplurality of workpieces is identified with the vision acquisitiondevice.
 13. The system of claim 11, wherein the generating the sequenceof the operations based on the one or more preconditions and the one ormore effects of the plurality of motion primitives, the target state,and the initial state of the plurality of workpieces comprises:generating the sequence of the operations in such a way that: everyprecondition of a start one of the operations meets the initial state ofthe plurality of workpieces; and every effect of a last one of theoperations meets the target state of the plurality of workpieces; andone or more effects of a previous one of the operations is capable ofchanging the plurality of workpieces from a previous state to asubsequent state, wherein one or more preconditions of a subsequent oneof the operations meet the subsequent state.
 14. The system of claim 11,wherein the method further comprises: performing a collision check todetermine whether the sequence of the operations is feasible; andresponsive to determining that the sequence of the operations is notfeasible: regenerating a new sequence of operations based on the one ormore preconditions and the one or more effects of the plurality ofmotion primitives, the target state, and the initial state of theplurality of workpieces.
 15. A non-transitory, computer readable mediumstoring instructions which, when executed by a processor, cause theprocessor to perform a method for controlling a robotic arm tomanipulate a plurality of workpieces, wherein the method comprises:predefining a plurality of motion primitives of the robotic arm, each ofthe plurality of motion primitives comprising one or more preconditionsand one or more effects; acquiring a target state of the plurality ofworkpieces, wherein the target state comprises relative connectionsbetween at least a subset of the plurality of workpieces; acquiring aninitial state of the plurality of workpieces, wherein the initial statecomprises connection types for at least the subset of the plurality ofworkpieces; determining if the connection types for at least the subsetof the plurality of workpieces are compatible with each other beforeperformance of operations; and generating a sequence of operations basedon the one or more preconditions and the one or more effects of theplurality of motion primitives, the target state and the initial stateof the plurality of workpieces, wherein generating the sequence of theoperations is performed in a way that every precondition of a startingone of the operations meets the initial state of the plurality ofworkpieces and every effect of a last one of the operations meets thetarget state of the plurality of workpieces; wherein the operations areselected from the plurality of motion primitives; wherein the initialstate of the plurality of workpieces further comprises locations of theplurality of workpieces, and relative positions and relative connectionsbetween the plurality of workpieces, and the sequence of operations isgenerated based on the locations of the plurality of workpieces, and therelative positions and the relative connections between the plurality ofworkpieces in the initial state; wherein execution of the sequence ofoperations is capable of changing the plurality of workpieces from theinitial state to the target state, including connecting at least thesubset of the plurality of workpieces according to the relativeconnections in the target state and the connection types.
 16. Thenon-transitory computer readable medium of claim 15, wherein thegenerating the sequence of the operations based on the one or morepreconditions and the one or more effects of the plurality of motionprimitives, the target state and the initial state of the plurality ofworkpieces comprises: generating the sequence of the operations in sucha way that: every precondition of a start one of the operations meetsthe initial state of the plurality of workpieces; and every effect of alast one of the operations meets the target state of the plurality ofworkpieces; and one or more effects of a previous one of the operationsis capable of changing the plurality of workpieces from a previous stateto a subsequent state, wherein one or more preconditions of a subsequentone of the operations meet the subsequent state.
 17. The non-transitorycomputer readable medium of claim 15, wherein a state of a prototypecorresponding to the target state of the plurality of workpieces isidentified, the state of the prototype comprising relative positions andrelative connections between workpieces of the prototype; the sequenceof operations is generated based on the relative positions and relativeconnections between workpieces of the prototype.